1. Data Loading and Preprocessing

1.1 Edge Data

1.2 Surface Data

1.3 Calibration Wafer Data

1.4 Calibration Surface Data

1.5 Witness Surface Data

2. Surface Area Distribution Analysis

3. Edge Area Distribution Analysis

4. Calibration Wafer Analysis - Edge

5. Calibration Surface Analysis

6. Witness Surface Analysis

6. Edge Model Analysis

7. Straight Line Model (SLSM)

## witness_surface_before9: 701 observations
## witness_surface_before10: 887 observations
## witness_surface_before11: 1295 observations
## witness_surface_before12: 214 observations
## witness_surface_before13: 1066 observations
## witness_surface_before14: 204 observations
## witness_surface_after9: 292467 observations
## witness_surface_after10: 144961 observations
## witness_surface_after11: 224730 observations
## witness_surface_after12: 121144 observations
## witness_surface_after13: 58732 observations
## witness_surface_after14: 99428 observations

##7.5

## === BRIGHTNESS ANALYSIS INITIALIZATION ===
## Analyzing image brightness patterns across surface and edge types...
## Target analyses: Starshade Surface, Calibration Surface, Witness Surface, Starshade Edge, Calibration Edge
## === SURFACE BRIGHTNESS ANALYSIS ===
## Available surface datasets: Starshade Surface Before, Starshade Surface After, Calibration Surface Before, Calibration Surface After, Witness Surface Before, Witness Surface After 
## Processing Starshade Surface Before - Total images: 15111 
## Processing Starshade Surface After - Total images: 15120 
## Processing Calibration Surface Before - Total images: 6146 
## Processing Calibration Surface After - Total images: 7631 
## Processing Witness Surface Before - Total images: 1413 
## Processing Witness Surface After - Total images: 3010 
## Total surface images analyzed: 48431 
## Images with transformation applied: 35462 
## # A tibble: 6 × 4
##   Surface_Type Timing Total_Images Transformed_Images
##   <chr>        <chr>         <int>              <int>
## 1 Calibration  After          7631               7631
## 2 Calibration  Before         6146               6146
## 3 Starshade    After         15120              10547
## 4 Starshade    Before        15111               8263
## 5 Witness      After          3010               2006
## 6 Witness      Before         1413                869
## === EDGE BRIGHTNESS ANALYSIS ===
## Available edge datasets: Starshade Edge, Calibration Edge 
## Processing Starshade Edge - Total samples: 975 
## Extracting trial numbers from column: image_name 
## Available brightness columns: Bef_Pre_Upper_Median, Bef_Pre_Lower_Median, Bef_Pre_Entire_Median, Bef_Post_Upper_Median, Bef_Post_Lower_Median, Bef_Post_Entire_Median, Aft_Pre_Upper_Median, Aft_Pre_Lower_Median, Aft_Pre_Entire_Median, Aft_Post_Upper_Median, Aft_Post_Lower_Median, Aft_Post_Entire_Median 
## Using brightness columns: Bef_Pre_Upper_Median, Bef_Pre_Lower_Median, Bef_Pre_Entire_Median, Bef_Post_Upper_Median, Bef_Post_Lower_Median, Bef_Post_Entire_Median, Aft_Pre_Upper_Median, Aft_Pre_Lower_Median, Aft_Pre_Entire_Median, Aft_Post_Upper_Median, Aft_Post_Lower_Median, Aft_Post_Entire_Median
## Processing Calibration Edge - Total samples: 994 
## Extracting trial numbers from column: image_name 
## Available brightness columns: Bef_Pre_Upper_Median, Bef_Pre_Lower_Median, Bef_Pre_Entire_Median, Bef_Post_Upper_Median, Bef_Post_Lower_Median, Bef_Post_Entire_Median, Aft_Pre_Upper_Median, Aft_Pre_Lower_Median, Aft_Pre_Entire_Median, Aft_Post_Upper_Median, Aft_Post_Lower_Median, Aft_Post_Entire_Median 
## Using brightness columns: Bef_Pre_Upper_Median, Bef_Pre_Lower_Median, Bef_Pre_Entire_Median, Bef_Post_Upper_Median, Bef_Post_Lower_Median, Bef_Post_Entire_Median, Aft_Pre_Upper_Median, Aft_Pre_Lower_Median, Aft_Pre_Entire_Median, Aft_Post_Upper_Median, Aft_Post_Lower_Median, Aft_Post_Entire_Median
## Total edge measurements analyzed: 23628 
## # A tibble: 2 × 3
##   Edge_Type   Total_Samples Total_Measurements
##   <chr>               <int>              <int>
## 1 Calibration           994              11928
## 2 Starshade             975              11700
## Creating surface brightness visualizations...
## Attempting timing effect analysis...
## Columns created after pivot_wider: Surface_Type, Trial 
## Expected columns found:  
## Could not find expected columns for timing effect analysis.
## Available columns: Surface_Type, Trial

## Creating edge brightness visualizations...

## 
## === SURFACE BRIGHTNESS SUMMARY STATISTICS ===
## # A tibble: 6 × 8
##   Surface_Type Timing Count Original_Mean Original_SD PostTransform_Mean
##   <chr>        <chr>  <int>         <dbl>       <dbl>              <dbl>
## 1 Calibration  After   7631           3.8         2.7                3.4
## 2 Calibration  Before  6146           4.8         2.2                3.5
## 3 Starshade    After  15120           2.1         1.6                2  
## 4 Starshade    Before 15111           1.6         1.6                1.5
## 5 Witness      After   3010           2           1.4                2  
## 6 Witness      Before  1413           1.8         1.5                1.8
## # ℹ 2 more variables: PostTransform_SD <dbl>, Avg_Change_Percent <dbl>
## 
## === SURFACE BRIGHTNESS NORMALIZATION EFFECTIVENESS ===
## # A tibble: 6 × 5
##   Surface_Type Timing Original_CV PostTransform_CV CV_Improvement_Percent
##   <chr>        <chr>        <dbl>            <dbl>                  <dbl>
## 1 Calibration  After        0.708            0.626                   11.5
## 2 Calibration  Before       0.463            0.632                  -36.5
## 3 Starshade    After        0.748            0.751                   -0.5
## 4 Starshade    Before       1.01             1.01                    -0.2
## 5 Witness      After        0.709            0.709                    0  
## 6 Witness      Before       0.791            0.791                    0
## 
## === EDGE BRIGHTNESS SUMMARY STATISTICS ===
## # A tibble: 24 × 8
##    Edge_Type Contamination Processing Region Count Mean_Brightness SD_Brightness
##    <chr>     <chr>         <chr>      <chr>  <int>           <dbl>         <dbl>
##  1 Calibrat… After Contam… After Pro… Entire   994            82.4          78.9
##  2 Calibrat… After Contam… Before Pr… Entire   994            93.6          89.1
##  3 Calibrat… Before Conta… After Pro… Entire   994           154.           60.3
##  4 Calibrat… Before Conta… Before Pr… Entire   994           170.           65  
##  5 Calibrat… After Contam… After Pro… Lower    994             8             0  
##  6 Calibrat… After Contam… Before Pr… Lower    994            10.5           2  
##  7 Calibrat… Before Conta… After Pro… Lower    994             8             0  
##  8 Calibrat… Before Conta… Before Pr… Lower    994            10.5           0.8
##  9 Calibrat… After Contam… After Pro… Upper    994           186             0  
## 10 Calibrat… After Contam… Before Pr… Upper    994           206.           14.9
## # ℹ 14 more rows
## # ℹ 1 more variable: CV <dbl>
## 
## === EDGE BRIGHTNESS CHANGE EFFECTS BY EDGE TYPE ===
## Average brightness change due to contamination:
## # A tibble: 12 × 5
##    Edge_Type   Processing   Region Mean_Contamination_E…¹ SD_Contamination_Eff…²
##    <chr>       <chr>        <chr>                   <dbl>                  <dbl>
##  1 Calibration After Proce… Entire                  -72.1                     NA
##  2 Calibration After Proce… Lower                     0                       NA
##  3 Calibration After Proce… Upper                     0                       NA
##  4 Calibration Before Proc… Entire                  -76.1                     NA
##  5 Calibration Before Proc… Lower                     0                       NA
##  6 Calibration Before Proc… Upper                     0.9                     NA
##  7 Starshade   After Proce… Entire                   -3.6                     NA
##  8 Starshade   After Proce… Lower                     0                       NA
##  9 Starshade   After Proce… Upper                     0                       NA
## 10 Starshade   Before Proc… Entire                   -0.7                     NA
## 11 Starshade   Before Proc… Lower                     1.1                     NA
## 12 Starshade   Before Proc… Upper                     5.8                     NA
## # ℹ abbreviated names: ¹​Mean_Contamination_Effect, ²​SD_Contamination_Effect
## 
## Average brightness change due to processing:
## # A tibble: 12 × 5
##    Edge_Type   Contamination  Region Mean_Processing_Effect SD_Processing_Effect
##    <chr>       <chr>          <chr>                   <dbl>                <dbl>
##  1 Calibration After Contami… Entire                  -11.3                   NA
##  2 Calibration After Contami… Lower                    -2.5                   NA
##  3 Calibration After Contami… Upper                   -20.2                   NA
##  4 Calibration Before Contam… Entire                  -15.2                   NA
##  5 Calibration Before Contam… Lower                    -2.5                   NA
##  6 Calibration Before Contam… Upper                   -19.2                   NA
##  7 Starshade   After Contami… Entire                   -3.4                   NA
##  8 Starshade   After Contami… Lower                    -3.8                   NA
##  9 Starshade   After Contami… Upper                    -5                     NA
## 10 Starshade   Before Contam… Entire                   -0.5                   NA
## 11 Starshade   Before Contam… Lower                    -2.6                   NA
## 12 Starshade   Before Contam… Upper                     0.8                   NA
## 
## === STARSHADE vs CALIBRATION EDGE COMPARISON ===
## # A tibble: 12 × 7
##    Contamination        Processing       Region Calibration Starshade Difference
##    <chr>                <chr>            <chr>        <dbl>     <dbl>      <dbl>
##  1 After Contamination  After Processing Entire        82.4     119.        37  
##  2 After Contamination  After Processing Lower          8        10          2  
##  3 After Contamination  After Processing Upper        186       200         14  
##  4 After Contamination  Before Processi… Entire        93.6     123.        29.1
##  5 After Contamination  Before Processi… Lower         10.5      13.8        3.3
##  6 After Contamination  Before Processi… Upper        206.      205.        -1.1
##  7 Before Contamination After Processing Entire       154.      123.       -31.6
##  8 Before Contamination After Processing Lower          8        10          2  
##  9 Before Contamination After Processing Upper        186       200         14  
## 10 Before Contamination Before Processi… Entire       170.      123.       -46.3
## 11 Before Contamination Before Processi… Lower         10.5      12.6        2.1
## 12 Before Contamination Before Processi… Upper        205.      199.        -6  
## # ℹ 1 more variable: Percent_Difference <dbl>
## 
## === BRIGHTNESS ANALYSIS COMPLETE ===
## Results stored in 'brightness_analysis_results' list for integration with main analysis.
## Surface types analyzed: Starshade Before, Starshade After, Calibration Before, Calibration After, Witness Before, Witness After
## Edge types analyzed: Starshade, Calibration
## 
## Available result components:
## - surface_data: Combined surface brightness data (Starshade, Calibration, Witness × Before/After)
## - edge_data: Combined edge brightness data (Starshade, Calibration)
## - surface_summary: Summary statistics for all surface types and timing
## - edge_summary: Summary statistics for all edge types
## - normalization_effectiveness: CV improvement metrics by surface type and timing
## - contamination_effects: Brightness changes due to contamination by edge type
## - processing_effects: Brightness changes due to processing by edge type
## - edge_comparison: Direct Starshade vs Calibration edge comparison
## - surface_plots: Surface visualization plots (distribution, change_by_transform, before_vs_after, variability, processing_effect, timing_effect, comparison)
## - edge_plots: Edge visualization plots (brightness_by_status, contamination_effect, processing_effect, edge_comparison)

8. Error Stats

8. Trial Anlysis Graphs

8.1 Trial 9

8.2 Trial 10

8.2 Trial 10 with Error Bars

8.3 Trial 11

8.4 Trial 12

8.5 Trial 13

8.6 Trial 14

9. Average Graphs

9.1 Average of Trials

9.2 Average Air of Trials

9.3 Scatter Graphs

9.4 Scatter Air Graphs

10. Comparison Graphics

10.1 Graph Comparison Graphic Count

## TableGrob (2 x 1) "arrange": 2 grobs
##   z     cells    name                 grob
## 1 1 (1-1,1-1) arrange text[GRID.text.5265]
## 2 2 (2-2,1-1) arrange      gtable[arrange]

10.2 Graph Comparison Graphic Scatter

## TableGrob (2 x 1) "arrange": 2 grobs
##   z     cells    name                 grob
## 1 1 (1-1,1-1) arrange text[GRID.text.6392]
## 2 2 (2-2,1-1) arrange      gtable[arrange]

10.3 Graph Comparison Graphic - Normalized Count

## 
## === MISSING DATA LOG ===
## Total missing data entries: 0
## No missing data detected - all datasets loaded successfully!
## 
## ========================

## TableGrob (2 x 1) "arrange": 2 grobs
##   z     cells    name                 grob
## 1 1 (1-1,1-1) arrange text[GRID.text.7504]
## 2 2 (2-2,1-1) arrange      gtable[arrange]

10.4 Normalized Scatter Graph

## Summary of missing data:
## Trial 9 - Surface Binned Scatter - ERROR: Error occurred: In argument: `valid_scatters = scatter_clean[is.finite(scatter_clean) & scatter_clean > 0]`. 
## Trial 9 - Edge Cumulative Scatter - ALL: No data lines available - using fallback 
## Trial 9 - Edge Binned Scatter - ALL: No data lines available - using fallback 
## Trial 10 - Surface Binned Scatter - ERROR: Error occurred: In argument: `valid_scatters = scatter_clean[is.finite(scatter_clean) & scatter_clean > 0]`. 
## Trial 10 - Edge Cumulative Scatter - ALL: No data lines available - using fallback 
## Trial 10 - Edge Binned Scatter - ALL: No data lines available - using fallback 
## Trial 11 - Surface Binned Scatter - ERROR: Error occurred: In argument: `valid_scatters = scatter_clean[is.finite(scatter_clean) & scatter_clean > 0]`. 
## Trial 11 - Edge Cumulative Scatter - ALL: No data lines available - using fallback 
## Trial 11 - Edge Binned Scatter - ALL: No data lines available - using fallback 
## Trial 12 - Surface Binned Scatter - ERROR: Error occurred: In argument: `valid_scatters = scatter_clean[is.finite(scatter_clean) & scatter_clean > 0]`. 
## Trial 12 - Edge Cumulative Scatter - ALL: No data lines available - using fallback 
## Trial 12 - Edge Binned Scatter - ALL: No data lines available - using fallback 
## Trial 13 - Surface Cumulative Scatter - ERROR: Error occurred: In argument: `valid_scatters = scatter_clean[is.finite(scatter_clean) & scatter_clean > 0]`. 
## Trial 13 - Surface Binned Scatter - ERROR: Error occurred: In argument: `valid_scatters = scatter_clean[is.finite(scatter_clean) & scatter_clean > 0]`. 
## Trial 13 - Edge Cumulative Scatter - ALL: No data lines available - using fallback 
## Trial 13 - Edge Binned Scatter - ALL: No data lines available - using fallback 
## Trial 14 - Surface Binned Scatter - ERROR: Error occurred: In argument: `valid_scatters = scatter_clean[is.finite(scatter_clean) & scatter_clean > 0]`. 
## Trial 14 - Edge Cumulative Scatter - ALL: No data lines available - using fallback 
## Trial 14 - Edge Binned Scatter - ALL: No data lines available - using fallback
## Plot list lengths:
## Surface cumulative: 8
## Surface binned: 8
## Edge cumulative: 8
## Edge binned: 8
## Required length: 8

## TableGrob (2 x 1) "arrange": 2 grobs
##   z     cells    name                 grob
## 1 1 (1-1,1-1) arrange text[GRID.text.8630]
## 2 2 (2-2,1-1) arrange      gtable[arrange]
## Grid created successfully!

11. Combined Analysis and Comparison

11.1 Cumulative Comparison

11.2 Binned Comparison

12. Multiple Line analysis

Instrument signature

## [1] "=== INSTRUMENT SIGNATURE GAP PREDICTION ==="
## [1] "Scale: 0.208333 microns/pixel"
## [1] "Area conversion: 0.043403 microns²/pixel²"
## [1] "\n=== SUMMARY COMPARISON ==="
## [1] "Raw Gap Counts (before merging nearby gaps):"
##           Scenario Threshold Circle_Gaps Bin_Gaps Threshold_Gaps Scale_Gaps
## 1   Starshade Edge        60          46        3              7          8
## 2 Calibration Edge        40          46        3              7          8
##   Raw_Total
## 1        64
## 2        64
## [1] "\nMerged Gap Counts (after combining gaps within 0.05 log units):"
##           Scenario Threshold Merged_Total Reduction Percent_Reduction
## 1   Starshade Edge        60           27        37              57.8
## 2 Calibration Edge        40           26        38              59.4
## [1] "\nGaps unique to Starshade Edge: 1"
## [1] "Gaps unique to Calibration Edge: 0"
## [1] "Starshade-specific gaps (first 5):"
## [1] 1.073
## [1] "\n=== ANALYSIS OF OBSERVED GAP ==="
## [1] "Observed gap at log10(area) = 2.3"
## [1] "Area: 199.5 μm²"
## [1] "Diameter: 15.9 μm"
## [1] "Closest Starshade prediction: 0.188 log units away"
## [1] "Closest Calibration prediction: 0.188 log units away"
## [1] "*** Starshade Edge model better predicts your observed gap ***"
## [1] "\n=== ARRAYS FOR RMARKDOWN ==="
## # Starshade Edge Predicted Gaps (Threshold 60)
## starshade_predicted_gaps <- c( 0.038, 0.092, 0.214, 0.339, 0.450, 0.533, 0.612, 0.691, 0.758, 0.884, 0.941, 0.992, 1.073, 1.155, 1.274, 1.328, 1.395, 1.457, 1.515, 1.569, 1.659, 1.736, 1.819, 1.895, 1.989, 2.081, 2.488 )
## # Calibration Edge Predicted Gaps (Threshold 40)
## calibration_predicted_gaps <- c( 0.038, 0.092, 0.214, 0.339, 0.436, 0.538, 0.612, 0.691, 0.758, 0.884, 0.941, 0.992, 1.155, 1.274, 1.328, 1.395, 1.457, 1.515, 1.569, 1.645, 1.747, 1.819, 1.895, 1.989, 2.081, 2.488 )
## [1] "\n=== VALIDATION READY ==="
## [1] "Use validate_gaps_simple() with your edge particle data:"
## [1] "starshade_validation <- validate_gaps_simple(edge_particles_data, 'Area', starshade_predicted_gaps)"
## [1] "calibration_validation <- validate_gaps_simple(calibration_particles_data, 'Area', calibration_predicted_gaps)"

13. Paper Comparisons

## === CHECKING AVAILABLE DATASETS ===
## Available datasets:
## - surface_binned : TRUE 
## - edge_binned : TRUE 
## - cw_edge_binned : TRUE 
## - model_edge_binned : TRUE
## Creating surface distribution plot...

## Surface distribution plot created successfully
## Creating edge distribution plot...

## Edge distribution plot created successfully
## Creating normalized distribution plot...

## Normalized distribution plot created successfully
## Creating ratio analysis plot...

## Ratio analysis plot created successfully
## === ANALYSIS SUMMARY ===
## Surface Distribution Analysis:
## - PCL: 175 μm²
## - Slope: -1.54 
## - Data points: 100
## Edge Distribution Analysis:
## - Total particles: 259 
## - Area range: 1 - 5987 μm²
## - Data points: 100
## Ratio Analysis:
## - Mean etched/calibration ratio: 2.56 
## - Median ratio: 0.71 
## - Maximum ratio: 31.48
## Plots successfully created: 4 out of 4
## ========================